Flight Sentry¶

Predicting Flight Departure Delays: A Time-Series Classification Approach¶

FP Phase 3 EDA, baseline pipeline, Scalability, Efficiency, Distributed/parallel Training, and Scoring Pipeline¶

1. Team 4-4 Introduction¶

1.1 Team Members:¶

1.2 Phase Leader Plan:¶

Because this is a team submission and each phase is graded separately, we’re assigning a weekly phase leader who is the single point of contact for that week. The leader is responsible for (i) keeping the notebook structure consistent, (ii) making sure the credit assignment table is updated, and (iii) pushing the HTML export on time. The work itself is still distributed across all five members, but the leader for that week owns the checklist and the delivery. The rotation below follows the course calendar (Phase 1 → Phase 2 → Phase 3) and also accounts for immersion/holiday weeks so we don’t miss the official B-Courses deadlines.

Current Phase Leader: Carlos Schrupp

2. Credit Assignment Plan:¶

2.1 Phase 1 Credit Assignment (Completed)¶

2.2 Phase 2 Credit Assignment (Completed)¶

2.3 Phase 3 Credit Assignment (Completed)¶

Evaluation | GPU experiments & threshold optimization | Ran GPU-based experiments for classifier threshold tuning, evaluated precision/recall/F2 trade-offs across operating points, documented results in dedicated notebook | Anabel | 6.0 | Evaluation | Performance metrics computation | Calculated accuracy, precision, recall, F1, F2, ROC-AUC for all models across all data splits | Shikha, Nicole, Carlos | 7.0 | | Evaluation | Feature importance analysis | Extracted top 15 features from RF, analyzed feature family contributions, validated diversity | Nicole, Shikha, Carlos | 5.0 | | Evaluation | Model comparison & gap analysis | Compared LR vs RF vs MLP, identified failure modes, documented pipeline limitations | Carlos | 6.0 | | Infrastructure | Pipeline stage documentation | Documented all 6 pipeline stages (S0-S5a) with row counts, feature counts, quality metrics | Arun | 5.0 | | Infrastructure | Computational benchmarking | Tracked wall times, cluster configurations for training and evaluation across all experiments | Carlos | 4.0 | | Phase 3 Total | | | | 426 |

2.4 Total Effort Summary¶

Individual Contribution Summary:

3. Project Structure:¶

3.1 Project Abstract¶

Abstract

Flight delays impose significant operational costs and customer dissatisfaction across the aviation industry. We are a data science startup tasked by an airline consortium to predict flight delays 2 hours before scheduled departure, enabling proactive management of crew scheduling, gate assignments, and passenger notifications. This project aims to predict whether a domestic U.S. flight will experience a departure delay of 15 minutes or more using information available at the 2-hour cutoff. Our hypothesis is that flight delays can be predicted using a combination of temporal patterns, weather conditions, airport congestion metrics, carrier performance history, network centrality, and time-series features available before departure. In Phase 1, we used the 3 Month OTPW (On-Time Performance and Weather) dataset, which combines flight data from the U.S. Department of Transportation's Bureau of Transportation Statistics with weather observations from the National Oceanic and Atmospheric Administration. Our Phase 2 analysis used the complete 2015 calendar year containing 5.7 million flights processed through a custom join and five-stage checkpoint pipeline that eliminates data leakage and achieves 0% missing data. Our Phase 3 analysis uses the complete 2015-2019 dataset containing 31.1 million flights processed through a six-stage pipeline that eliminates most data leakage, custom joins the data, and achieves near-zero missing data. The dataset exhibits an 82:18 class imbalance with 82% on-time flights. Our feature engineering and selection pipeline produced 112 optimized features across 8 families: Rolling Aggregates (18), Temporal Features (24), Weather Features (15), Network/Graph Features (8), RFM Features (12), Interaction Terms (16), Meta-Features (4), and Binary Indicators (22), all strictly using information available at least 2 hours before scheduled departure.

We selected Logistic Regression as our interpretable baseline, which provides fast training on distributed infrastructure, well-calibrated probability outputs for threshold tuning, and serves as an industry-standard benchmark. To address the 82:18 class imbalance, we applied undersampling to the training set while preserving natural class distributions in validation (2018) and test (2019) sets. We optimized for F₂-score, which weights recall higher than precision to prioritize catching risky flights—aligning with airline operational priorities. The baseline Logistic Regression with non-engineered features achieved AUC-PR of 0.515 on the 2019 blind holdout. Feature engineering substantially improved performance, with Logistic Regression on engineered features achieving F₂ of 0.595, precision of 45.4%, and recall of 64.5%. Random Forest (Phase 3) reached F₂ of 0.610 with precision of 64.8% and recall of 60.1%, while identifying the top predictive features: 24-hour weighted rolling average delay by origin airport (14.2% importance), Random Forest probability meta-feature (11.8%), previous flight delay status (9.5%), origin degree centrality (8.7%), and prior-day delay rate (7.6%). The best-performing model was MLP 40:60 with optimized two-layer architecture, achieving F₂ of 0.662, precision of 36.7%, and recall of 82.9%, demonstrating superior ability to identify at-risk flights for operational intervention. For delay duration prediction, our two-tier regression model achieved RMSE of 42.83 minutes on the 2019 holdout, providing actionable minute-level estimates for operational planning. The diversity across our top 15 features—spanning rolling aggregates, meta-features, aircraft lag variables, and network centrality—validates that delays are complex phenomena requiring multiple analytical perspectives.

Phase 3 accomplishments include: scaling the pipeline 6x from 5M to 31M flights while maintaining data quality, engineering 112 leakage-free features validated against the T-2 hour cutoff, implementing and comparing four modeling approaches (Logistic Regression baseline, Random Forest classification, MLP neural networks, and two-tier regression), establishing rigorous train-validation-holdout splits (2015-2017 train, 2018 validation, 2019 blind test), and achieving production-ready data quality with less than a percent of missing values. Next steps for deployment include: implementing real-time feature computation infrastructure, developing data drift monitoring for model degradation detection, creating carrier-specific threshold calibration for heterogeneous operational needs, and validating pipeline performance on 2020-2021 data to assess COVID-19 impact and model robustness to unprecedented disruptions.

3.2 Project Description¶

This report documents the development of our machine learning pipeline for predicting flight departure delays across three project phases, with Phase 3 delivering a production-ready, multi-year system evaluated on 31.1 million flights (2015-2019) with 112 optimized features and multiple modeling approaches including neural networks.

Report Structure:

• Section 3.3 - The Data: Details our production dataset (31.1M flights, 112 features from 2015-2019), the six-stage checkpoint pipeline from raw ingestion to modeling-ready format (3.3.3), comprehensive data quality analysis achieving <1% missing data (3.3.4), and target variable analysis with class imbalance (18.15% delayed, 82:18 ratio) in Section 3.3.5. Section 3.3.7 documents the custom flights-weather-airport join architecture rebuilt for 5-year data with feature leakage prevention strategy. Sections 3.3.8-3.3.9 cover feature engineering methods across 8 families and the complete feature dictionary. Section 3.3.6 presents embedded EDA findings on temporal delay patterns, carrier performance, and geographic patterns across all five years. Sections 3.3.10-3.3.11 detail dataset storage requirements and production readiness validation.

• Section 4 - Exploratory Data Analysis: Provides comprehensive multi-year visualizations and analysis including data overview (4.1), time-based delay patterns with hourly and day-of-week breakdowns (4.2.1-4.2.2), airport-specific delay patterns showing volume vs performance relationships (4.3), airline performance comparisons across market share and delay rates (4.4), and weather impact assessments covering temperature, wind speed, precipitation effects (4.5.1-4.5.3), visibility impacts (4.6), and correlation heatmap analysis (4.7).

• Section 5 - Machine Learning Algorithms and Modeling Strategy: Describes our modeling approach (5.1.1) with time-series cross-validation (5.1.3). Section 5.2 presents classification baselines including Logistic Regression (5.2.1.1-5.2.1.3), Random Forest (5.2.1.4), and comprehensive classification results comparison (5.2.1.5-5.2.1.6). Regression baseline analysis (5.2.2) includes feature summaries (5.2.2.2), performance results (5.2.2.3), holdout evaluation (5.2.2.4.2), feature importance by model (5.2.2.5), and regression summary (5.2.2.7). Section 5.3 details the alternative two-stage architecture with holdout results, error analysis by delay magnitude, worst-performing airports/carriers/routes, weekend vs holiday comparisons, and feature importance comparisons between classifier and regressor. Section 5.4 presents MLP neural network ensemble implementation including data splits (5.4.1), leakage control (5.4.2), feature engineering pipeline (5.4.3), class imbalance diagnosis (5.4.4), ensemble member construction with 50:50 undersampling (5.4.5), Optuna hyperparameter optimization (5.4.6), final model training (5.4.7), inference methods (5.4.8), results summary comparing single models vs ensemble (5.4.11), confusion matrix analysis (5.4.12), and saved artifacts (5.4.13). Section 5.5 summarizes all classification experiments, and Section 5.6 discusses time-series, graph, and MLP implementations.

• Section 6 - Evaluation Metrics: Establishes F₂-score as primary classification metric (6.1) prioritizing recall over precision to catch at-risk flights—aligning with airline operational priorities where missing a delay causes more disruption than a false alarm, with PR-AUC as secondary metric (6.2) appropriate for imbalanced datasets. Includes per-segment confusion analysis by carrier and airport (6.3), regression metrics MAE and RMSE for Stage 2 duration prediction (6.4), and operational/domain-level evaluation views (6.5).

• Section 7 - Gantt Chart: Presents project timeline tracking parallel workstreams across three phases including data engineering tasks, feature development activities, baseline and advanced modeling efforts, MLP implementation, and report preparation milestones.

• Section 8 - Pipeline: Illustrates end-to-end six-stage Spark ML pipeline with explicit checkpoints, from raw data ingestion (Stage 0) through leakage-aware cleaning and joins (Stage 1-2), time-series split and feature engineering using only T-2h information (Stage 3-4), feature selection and final refinement (Stage 5-5a), model training with undersampled data, and evaluation using time-ordered validation with 2019 blind holdout.

• Section 9 - Conclusion: Summarizes key findings, model performance across three approaches, feature importance insights, and project outcomes demonstrating production readiness.

• Section 10 - Open Issues and Next Steps: Discusses deployment considerations including real-time feature computation infrastructure, data drift monitoring for model degradation detection, carrier-specific threshold calibration, and validation on 2020-2021 data to assess COVID-19 impact and model robustness.

• Appendix: Provides references and links to supporting Databricks notebooks for data cleaning, EDA, custom joins, feature engineering, and modeling pipelines, plus academic references on flight delay prediction methodologies.

Key Technical Achievements:

Throughout this project, we emphasize reproducibility through checkpoint-based workflows, scalability using Apache Spark's distributed computing (validated on 31.1M records with cluster benchmarking), and strict adherence to temporal validation protocols that prevent look-ahead bias. Our T-2h prediction window reflects realistic operational conditions where airlines must make crew scheduling, gate assignment, and passenger notification decisions before actual departure information becomes available. The six-stage pipeline successfully handles the 6× data scale increase through strategic caching, checkpointing, and unpersisting operations. Feature importance analysis across Random Forest models identified the top predictors: 24-hour weighted rolling average delay by origin airport (14.2%), RF probability meta-feature (11.8%), previous flight delay status (9.5%), origin degree centrality (8.7%), and prior-day delay rate (7.6%). The diversity across our top 15 features—spanning rolling aggregates, meta-features, aircraft lag variables, and network centrality—validates that delays are complex phenomena requiring multiple analytical perspectives drawn from temporal, operational, environmental, and systemic factors.

Evolution from Phase 2 to Phase 3:¶

3.3 The Data¶

3.3.1 Data Sources¶

Our flight delay prediction system integrates five primary data sources, each serving a distinct purpose in building a comprehensive modeling dataset.

Table 3.1: Data Sources Overview

For Phase 2, we built a custom leakage-safe join rather than using the pre-joined OTPW dataset. This approach enabled precise T-2h weather alignment, better airport metadata integration, and complete control over the join logic to prevent data leakage. In Phase 3, we scaled this custom join infrastructure to handle five years of data (2015-2019), rebuilding all weather joins, aircraft rotations, and airport metadata joins for 31.1 million flights while maintaining strict temporal ordering and leakage prevention.

3.3.2 Dataset Scope and Dimensions¶

Phase 3 Analysis Dataset¶

For Phase 3, we conduct comprehensive analysis on the complete 2015-2019 dataset, representing five full years of domestic U.S. flight operations. This provides complete temporal coverage to capture seasonal patterns, year-over-year trends, operational dynamics, holiday effects, and weather variability across multiple annual cycles—essential for building a production-ready prediction system.

Table 3.2: Final Dataset Dimensions (Checkpoint 5a)

Yearly Distribution:

• 2015: 5,704,114 flights (18.3%)
• 2016: 5,518,291 flights (17.7%)
• 2017: 5,604,527 flights (18.0%)
• 2018: 7,137,918 flights (22.9%)
• 2019: 7,328,857 flights (23.5%)

Train-Validation-Test Split:

• Training Set (2015-2017): 16,826,932 flights (54.0%)
• Validation Set (2018): 7,137,918 flights (22.9%)
• Blind Holdout (2019): 7,328,857 flights (23.5%)

This temporal split ensures no data leakage while providing sufficient training data and maintaining realistic operational evaluation on unseen future years.

Scalability Achievements¶

We successfully scaled our pipeline 6× from Phase 2's single-year dataset to five years of data by implementing aggressive memory management strategies including:

• Checkpoint-based workflows: Intermediate results saved at each pipeline stage to enable recovery and avoid recomputation
• Strategic caching: Frequently-accessed DataFrames cached in memory with explicit unpersisting when no longer needed
• Parquet optimization: Converted all intermediate and final datasets to columnar Parquet format for improved compression and query performance
• Partition tuning: Adjusted Spark partitions based on data volume at each stage to balance parallelism and overhead

Dataset Scale Progression:

• 3-month sample: ~1.4M flights (Phase 1 prototyping)
• 1-year dataset: ~5.7M flights (Phase 2 production baseline)
• 5-year dataset: ~31.1M flights (Phase 3 production-ready system)
• 2020-2024 dataset: ~31.3M flights (Phase 3 extra points)

We read data directly into Apache Spark using spark.read.parquet() for all checkpointed stages, leveraging Spark's lazy evaluation and distributed processing. Cluster configurations were scaled from 4-node to 8-node setups during intensive feature engineering stages, with wall times tracked for all major pipeline operations to ensure computational feasibility.

3.3.3 Data Processing Pipeline¶

Our pipeline implements a systematic, six-stage transformation workflow designed to convert raw flight and weather data into a production-ready modeling dataset. Each stage is checkpointed to enable reproducibility, debugging, and collaborative development.

Table 3.3: Pipeline Stage Summary (2015-2019)

Key Transformations by Checkpoint¶

Stage 0 → CP1: Weather and Geographic Join (+73,722 rows, -139 columns)

• Custom T-2 hour aligned join with NOAA weather stations (634 locations)
• Added 15 weather features (temperature, wind, precipitation, visibility, pressure, humidity)
• Added 20 geographic features (airport coordinates, station distances, timezones, elevation)
• Consolidated/removed 139 redundant OTPW columns
• Row increase due to join expansion (flights matched to multiple weather observations)
• Missing data reduced: 49.39% → 10.16%

CP1 → CP2: Data Cleaning and Leakage Removal (-617,950 rows, -16 features)

• Removed 15 data leakage features containing future information:
  • Actual times: DEP_TIME, ARR_TIME, WHEELS_OFF, WHEELS_ON, TAXI_OUT, TAXI_IN, ACTUAL_ELAPSED_TIME, AIR_TIME
  • Delay breakdowns: CARRIER_DELAY, WEATHER_DELAY, NAS_DELAY, SECURITY_DELAY, LATE_AIRCRAFT_DELAY
  • Target leakage: ARR_DEL15, ARR_DELAY
• Filtered cancelled flights (28,812 records) and diverted flights (589,138 records) as operationally distinct scenarios
• Applied 3-tier weather imputation strategy:
  • Tier 1: Actual observed value
  • Tier 2: 24h rolling average by airport
  • Tier 3: Global median
• Result: 10.16% missing → 0.00% missing
• All 477,296 target variable nulls removed

CP2 → CP3: Basic Feature Engineering (+36 features)

• Temporal Features (10): departure_hour, departure_month, departure_dayofweek, is_weekend, season, is_peak_hour, time-of-day categories, day/hour interaction
• Distance Features (6): log_distance, distance bins (very_long, etc.), DISTANCE_high_corr, DISTANCE_GROUP_high_corr
• Weather Features (3): weather_condition_category, temp_anomaly, sky_condition_parsed
• Rolling Metrics (10): rolling_origin_num_delays_24h, rolling_origin_delay_ratio_24h, dep_delay15_24h_rolling_avg variants by origin/carrier/dayofweek
• Geographic (3): airport_traffic_density, carrier_flight_count
• Other (4): is_superbowl_week, is_major_event, is_airport_maintenance, is_natural_disaster

CP3 → CP4: Advanced Feature Engineering (+91 features)

• Weather Expansion (+19): Hourly wind direction, gust speed, relative humidity, station pressure, altimeter setting, additional weather composites
• Rolling Features (+12): 30-day route delay rates, carrier delays at origin, same-day statistics, time-based congestion ratios
• RFM Features (13): days_since_last_delay_route, days_since_carrier_last_delay_at_origin, route_delays_30d, route_delay_rate_30d, carrier_delays_at_origin_30d, 1-year delay rates
• Interaction Terms (24): peak_hour × traffic, weekend × route_volume, weather × airport_delays, temp × holiday, carrier × hour, origin × weather/visibility/precipitation/wind, origin × dest, carrier × origin/dest
• Cyclic Encoding (14): dep_time_sin/cos, arr_time_sin/cos, day_of_week_sin/cos, month_sin/cos (7 pairs = 14 features)
• Network Features (8): origin_degree_centrality, dest_betweenness, delay_propagation_score, network_delay_cascade, origin_1yr_delay_rate, dest_1yr_delay_rate, dest_delay_rate_today
• Aircraft Lag (6): prev_flight_dep_del15, prev_flight_crs_elapsed_time, hours_since_prev_flight, turnaround_category, num_airport_wide_delays, oncoming_flights
• Breiman Meta-Features (2): rf_prob_delay, rf_prob_delay_binned

CP4 → CP5: Feature Optimization and Reduction (-33 features)

• Removed high-correlation features (Pearson >0.85): YEAR, QUARTER, DISTANCE_high_corr, DISTANCE_GROUP_high_corr, and 29 others
• Removed high-cardinality features: TAIL_NUM, FL_DATE, prediction_utc, origin_obs_utc, asof_minutes
• Removed duplicate features identified through analysis
• Removed low-importance features with zero contribution in preliminary models
• Result: 186 features → 153 features (18% reduction)

CP5 → CP5a: String Indexing and Final Selection (-41 features, +12 indexed)

• String Indexing (12 categorical features converted):
  • DEST_indexed, ORIGIN_indexed, OP_UNIQUE_CARRIER_indexed
  • ORIGIN_STATE_ABR_indexed, DEST_STATE_ABR_indexed
  • origin_type_indexed, season_indexed, weather_condition_category_indexed
  • airline_reputation_category_indexed, turnaround_category_indexed
  • day_hour_interaction_indexed, sky_condition_parsed_indexed
• Removed 41 features: Original string columns retained for validation but removed from modeling set, plus additional low-importance features identified through feature importance analysis
• Final validation: Verified <1% missing data (0.01%), confirmed zero leakage, validated all data types
• Result: 153 features → 112 production-ready features (27% reduction from CP4)

Checkpoint Strategy¶

All checkpoints are stored at dbfs:/student-groups/Group_4_4/ with the following benefits:

• Resumability: Team members can start from any intermediate stage without re-running 6+ hour join operations on 31M records
• Debugging: Errors can be traced to specific pipeline stages with detailed logging at each checkpoint
• Collaboration: Multiple team members can work on different feature families simultaneously
• Version Control: Each checkpoint represents a stable, validated state of data transformation
• Computational Efficiency: Expensive operations (weather joins, rolling aggregations, network metrics) are computed once and cached
• Memory Management: Strategic caching, checkpointing, and unpersisting at each stage prevents out-of-memory errors on 31M+ row dataset
• Quality Assurance: Each checkpoint includes validation checks for missing data, duplicates, leakage, and data type consistency

3.3.4 Data Quality and Missing Data Analysis¶

Quality Metrics Across Pipeline¶

Our systematic data processing pipeline achieved comprehensive quality improvement, reducing missing data from 49.39% to <1% while preserving 98.3% of flight records from the raw OTPW source and ensuring zero data leakage.

Table 3.4: Data Quality Metrics Across Pipeline Stages (2015-2019)

Row Count Evolution¶

Stage 0 → CP1: Join Expansion (+73,722 rows, +0.2%)

• Raw OTPW contained 31,673,119 flight records
• Weather join added 73,722 rows due to multiple weather station matches per airport
• Some flights matched to backup weather stations when primary station data was unavailable
• This temporary expansion is expected behavior for left join with geographic proximity matching

CP1 → CP2: Data Cleaning (-617,950 rows, -1.9%)

• Total rows removed: 617,950 (1.95% of CP1)
• Breakdown:
  • Cancelled flights: ~141,000 records (estimated 0.44% of CP1)
  • Diverted flights: ~98,000 records (estimated 0.31% of CP1)
  • Target nulls (no delay status): 477,296 records (1.50% of CP1, includes overlap with cancelled/diverted)
  • Invalid records (missing critical fields): remaining balance
• Final retention: 31,128,891 rows (98.05% of CP1, 98.28% of Stage 0)

CP2 → CP5a: Feature Engineering (0 rows removed)

• All feature engineering stages preserved 100% of cleaned records
• Focus shifted from row filtering to column expansion and optimization
• Strict preservation ensures reproducibility and auditability

Cancelled and Diverted Flights¶

Cancelled and diverted flights were filtered from the dataset during CP2 cleaning:

• Cancelled flights: ~141,000 records (0.44% of CP1, 2015-2019)
• Diverted flights: ~98,000 records (0.31% of CP1, 2015-2019)
• Total removed with nulls: 617,950 records (1.95% of CP1)

Why Remove These Flights?

Cancelled and diverted flights represent fundamentally different operational scenarios than delayed departures:

1. Cancelled flights never depart, making departure delay prediction undefined. Including them would require predicting whether a flight will be cancelled (a different binary task) rather than whether a departing flight will be delayed.

2. Diverted flights involve mid-flight operational decisions (weather, medical emergencies, mechanical issues) that cannot be predicted at the T-2 hour departure window. Their "departure delay" may be minimal, but they don't arrive at the intended destination.

3. Target variable integrity: DEP_DEL15 is undefined or irrelevant for cancelled flights (no departure occurred) and misleading for diverted flights (successful departure but operational failure).

By filtering these records, we ensure our model focuses exclusively on flights that departed as scheduled to their intended destination, making predictions directly actionable for operational decision-making around crew scheduling, gate assignments, and passenger notifications.

Missing Data Reduction Strategy¶

Our three-tier imputation approach systematically eliminated missing data across 5 years:

Stage 0: Raw OTPW

• 142 of 214 columns contained missing values
• Primary sources: weather features not yet joined, optional OTPW fields
• 475,789 flights missing DEP_DEL15 target variable

CP1: After Weather Join

• Weather join reduced missing data significantly
• 51 of 75 columns still contained missing values
• Remaining gaps: weather observations outside T-2 hour window, station outages
• 477,296 flights still missing DEP_DEL15

CP2: Three-Tier Imputation

• Tier 1 (Preferred): Use actual observed weather value from matched station
• Tier 2 (Historical): Use 24-hour rolling average by airport if no observation within T-2 window
• Tier 3 (Global): Use global median for weather variable if no historical data available
• Categorical nulls: Replace with 'UNK' indicator to preserve missingness signal
• Target variable: Remove all 477,296 records with null DEP_DEL15 (overlap with cancelled/diverted)
• Result: Complete elimination of missing values across all 59 columns

CP3-CP4: Minimal Missing from New Features

• Basic features (CP3): No missing values (all derived from complete CP2 data)
• Advanced features (CP4): Minimal missing in 6 RFM features
  • Missing occurs for first flights on routes (no historical delay data)
  • Represents edge case: route's inaugural flight or first flight after long gap
  • Strategy: Impute with carrier-average or route-average from similar distances

CP5-CP5a: Production Quality

• CP5: Minimal missing in 4 features (same_day_prior_delay_percentage, dest_delay_rate_today, route_delays_30d, carrier_delays_at_origin_30d)
• CP5a: Less than 1% missing maintained (563,214 nulls in same_day_prior_delay_percentage = 1.81% of rows)
• Strategic decision: Retain minimal missing in temporal features as signal (first flight of day, new route)
• Modeling approach: Tree-based models handle these naturally; for linear models, impute with 0 or create is_missing indicator

Data Quality Validation Checks¶

Checkpoint 2 Validation (Post-Cleaning):

• Zero missing values in core features (ORIGIN, DEST, OP_UNIQUE_CARRIER, FL_DATE)
• Zero missing values in target variable (DEP_DEL15)
• All weather features imputed (15 features complete)
• All geographic features complete (20 features)
• No duplicate records (verified via FL_DATE + OP_CARRIER_FL_NUM + ORIGIN + DEST)
• Date range validated: 2015-01-01 to 2019-12-31 (no out-of-range dates)
• All numeric features within expected ranges (temperature: -50°F to 130°F, etc.)

Checkpoint 5a Validation (Production-Ready):

• 99.99% data completeness (0.01% missing in 4 temporal features by design)
• All categorical features indexed (12 indexed columns)
• Zero data leakage (all features validated against T-2 hour cutoff)
• All date/time features in UTC (timezone consistency)
• Target variable complete with valid class balance (81.85% / 18.15%)
• No high-correlation features remaining (Pearson >0.85 removed)

Missing Data Breakdown by Feature Category (CP1)¶

After the initial weather join (CP1), missing data was reduced from 49.39% (Stage 0) to 10.16%, concentrated in specific feature categories:

Table 3.5: Missing Data by Category (CP1 - Post-Weather Join)

Understanding Weather Data Missingness¶

Weather observations are event-driven rather than continuous. NOAA stations report certain conditions only when they occur, making missingness informative:

• Wind gusts are only measured when gusts exceed sustained wind speed thresholds. Null values typically indicate calm conditions rather than missing observations.

• Present weather types (rain, fog, thunderstorm codes) are only logged during active weather events. Clear, calm weather often results in null entries rather than explicit "clear" codes.

• Precipitation amounts are null during dry periods rather than recorded as zero. This follows meteorological convention where absence of measurement indicates absence of precipitation.

• Remote airports have limited weather station coverage, leading to temporal gaps where the nearest station is beyond the reliable proximity threshold.

This pattern means that missing weather data often indicates benign conditions rather than data quality issues. Our three-tier imputation strategy accounts for this by using historical patterns at the same airport before falling back to global medians.

Imputation Results¶

Features Requiring Imputation (15 weather features):

• Temperature: HourlyDryBulbTemperature, HourlyDewPointTemperature, HourlyWetBulbTemperature
• Precipitation: HourlyPrecipitation
• Wind: HourlyWindSpeed, HourlyWindDirection, HourlyWindGustSpeed
• Visibility: HourlyVisibility
• Atmospheric: HourlyRelativeHumidity, HourlyStationPressure, HourlySeaLevelPressure, HourlyAltimeterSetting
• Conditions: HourlyPresentWeatherType, HourlySkyConditions, HourlyDryBulbTemperature

Imputation Coverage (CP2):

• Tier 1 (Actual observed): Approximately 90% of values used as-is
• Tier 2 (24h rolling average): Approximately 8% imputed from historical airport data
• Tier 3 (Global median): Approximately 2% imputed from overall distribution
• Result: 10.16% missing (CP1) → 0.00% missing (CP2)

3.3.5 Target Variable and Class Balance¶

Target Definition¶

Our binary classification task predicts DEP_DEL15, where:

• DEP_DEL15 = 1: Flight delayed ≥15 minutes from scheduled departure
• DEP_DEL15 = 0: Flight on-time (delayed <15 minutes)

This 15-minute threshold aligns with the U.S. Department of Transportation's official definition of flight delay used in carrier performance reporting.

Table 3.5: Target Variable Distribution

Imbalance Ratio: 4.44:1 (on-time : delayed)

Class Imbalance Mitigation Strategies¶

The 4.44:1 imbalance is substantial but manageable through multiple complementary approaches:

1. SMOTE Undersampling (Current Approach): We apply undersampling of the majority class (on-time flights) to training data only, creating a more balanced training set while preserving the original test distribution for realistic evaluation.

2. Class Weighting (Previously Tested): We initially experimented with inverse frequency weights during model training (weight_0 = 1.0, weight_1 = 4.44), but found SMOTE undersampling provided better results for our use case.

3. Threshold Tuning: Adjust decision threshold from default 0.5 to optimize F₀.₅-score based on precision-recall trade-offs.

4. Ensemble Methods: Tree-based models (Random Forest, Gradient Boosting) naturally handle imbalance via sample weighting and bagging.

5. Evaluation Metrics: Prioritize F₀.₅-score, precision, and precision-recall AUC over accuracy to avoid majority-class bias.

Business Justification: From a business perspective, false negatives (predicting on-time when actually delayed) are more costly than false positives. Airlines can proactively notify passengers, adjust crew scheduling, and optimize aircraft rotation when delays are predicted, even if some predictions are false alarms. Therefore, we optimize for high recall on the delayed class.

3.3.6 Data Results - Exploratory Data Analysis¶

Our EDA reveals critical patterns in temporal dynamics, carrier performance, geographic distributions, and distance effects that inform feature engineering and modeling strategies.

Temporal Patterns¶

Table 3.6: Key Temporal Insights (2015-2019 Analysis)

Overall Dataset Characteristics:

• Total flights analyzed: 31,128,891 (2015-2019)
• Overall delay rate: 18.15%
• Best performing day: Saturday (16.21% delay rate)
• Worst performing day: Friday (19.74% delay rate)
• Lowest delay quarter: Q4 (16.30%)
• Highest delay quarter: Q2 (19.39%)

Carrier Performance¶

Table 3.7: Key Carrier Insights

Overall Carrier Characteristics:

• Total carriers analyzed: 19 distinct operators (2015-2019)
• Overall delay rate: 18.03%
• Best performing carrier: Hawaiian Airlines (HA) - 7.61% delay rate
• Worst performing carrier: JetBlue (B6) - 25.01% delay rate
• Largest carrier by volume: Southwest (WN) - 6.5M flights (20.8% market share)
• Most efficient large carrier: Delta (DL) - 4.6M flights with 14.24% delay rate
• Carrier performance spread: 17.4 percentage points (HA to B6)

Geographic Patterns¶

Table 3.8: Key Geographic Performance Insights (2015-2019 Analysis)

Overall Geographic Characteristics:

• Total states analyzed: 53 (all US states + territories, 2015-2019)
• Total unique airports: 369 origins, 368 destinations
• State delay rate mean: ~18%
• State delay rate median: ~18%
• Highest volume state: California (CA) - ~3.5M flights
• Worst performing airports: MDW, DAL, EWR, FLL, LGA (~23-24% delay rates)
• Best performing major airport: Honolulu (HNL) - ~13% delay rate
• Northeast corridor delay premium: 2-5pp above national average
• Airport delay spread: ~11 percentage points across top performers

Distance and Flight Duration¶

Table 3.9: Distance and Duration Analysis (2015-2019)

Weather Impact Patterns¶

Correlation analysis between weather features and delays from 5-year dataset:

Table 3.10: Weather Feature Correlations with DEP_DEL15

Key Weather Insights:

• Individual weather features show weak to moderate correlations (-0.04 to +0.05), suggesting weather is a contributing but not dominant factor
• Extreme values matter more than means: temperatures below 25°F and wind gusts above 30 units show disproportionate delay impact
• Composite weather features (weather_condition_category, temp_anomaly) capture non-linear relationships better than raw measurements
• Weather × airport interactions are critical: same weather conditions affect airports differently based on infrastructure and operational capacity
• Emphasizes need for comprehensive feature engineering capturing operational factors (aircraft history, airport congestion, time-of-day, carrier performance) alongside weather conditions

3.3.7 Custom Flights to Weather Join¶

Entity-Relationship Model¶

The following diagram illustrates the relationships between our core data entities used for enriching flight records with meteorological data:

Data Integration Challenges¶

Our exploratory review identified key blockers in the lookup tables rather than the flights themselves. These challenges were addressed once in Phase 2 and scaled to 31M records in Phase 3:

Table 3.11: Data Integration Challenges and Solutions

As-Of Join Logic¶

Our custom, leakage-safe join combines DOT on-time performance with NOAA Global Hourly weather at the origin airport as-of T-2h before scheduled departure. This join was executed on 31.7M raw OTPW records, producing 31.7M joined records in CP1.

Join Algorithm (T-2 Hour Weather Alignment):

1. Normalize flight times using master airports dimension (IATA, timezone, lat/lon for 369 airports)
2. Convert scheduled departure to UTC for weather alignment: prediction_utc = scheduled_dep_local - 2h → UTC
3. Link airports to nearest 3 NOAA stations via haversine distance (634 total stations, average 2.1 stations/airport)
4. Filter to valid hourly report types (FM-15, FM-16, FM-12) from NOAA ISD format
5. Apply strict as-of rule: obs_utc ≤ prediction_utc with 6-hour lookback window
6. Select latest qualifying observation, preferring station rank 1 → 2 → 3 based on proximity
7. Preserve provenance fields: station_id, origin_obs_utc, origin_station_dis, asof_minutes
8. Select only T-2h valid features: visibility, wind speed/direction/gusts, precipitation, temperature, humidity, pressure, sky conditions
9. Exclude all leakage features: actual departure/arrival times, taxi times, delay cause breakdowns, cancelled/diverted status

Join Statistics (2015-2019):

• Input rows (Stage 0 OTPW): 31,673,119
• Output rows (CP1): 31,746,841 (+73,722 rows, +0.2%)
• Row expansion: Some flights matched to multiple backup weather stations when primary station had missing observations
• Weather match rate: ~99.8% of flights successfully matched to at least one weather observation within 6-hour window
• Average asof_minutes: ~45 minutes (weather observation typically 30-60 minutes before T-2h cutoff)
• Missing weather features after join: 10.16% (down from 49.39% in Stage 0)
• Stations utilized: 634 NOAA stations across US
• Airports covered: 369 origin, 368 destination

Performance Optimizations for 31M Records:

• Airport dimension broadcast join (small lookup table ~500 rows)
• Weather data partitioned by year and month for efficient temporal filtering
• Station bridge table cached (airport-to-station mappings computed once)
• Haversine distance calculations vectorized using PySpark UDFs
• Checkpoint after join to prevent recomputation (CP1 saved)
• Processing time: ~6 hours on 8-node cluster

Phase 2 Deliverables:

• Custom join notebook implementing T-2h weather alignment
• Airport dimension table with timezone/coordinate enrichment
• Airport-to-station bridge table with haversine distances
• Join validation notebook verifying leakage-free properties
• Scalable to full 2015-2019 range using same pipeline logic

Phase 3 Enhancements:

• NOAA and BTS data downloader notebook for acquiring raw 2015-2019 data
• Schema standardization notebook ensuring consistent column names/types across years
• Upgraded automatic join notebook processing 31.7M records with checkpointing
• Full 2015-2019 dataset processed through join pipeline
• Validation of join statistics and weather coverage across all 5 years
• Documentation of memory optimization strategies for 6x data scale increase

Reproducibility: All join logic, airport mappings, and station bridges are versioned and checkpointed, enabling reproducible processing of future years (2020+) with identical methodology.

Feature leakage prevention strategy¶

Data Join Summary¶

Table 3.10: Data Join Summary (Flights × Weather)

3.3.8 Feature Engineering¶

Our feature engineering process transformed 59 cleaned features (CP2) into 112 production-ready features (CP5a) through systematic addition, transformation, and selection across four pipeline stages. Each transformation method was chosen to capture domain-specific flight delay patterns while maintaining strict adherence to the T-2 hour prediction cutoff.

Feature Transformation Overview¶

Table 3.11: Feature Transformation Methods (2015-2019)

Dimensionality Reduction Approach¶

Our feature selection process employed multiple statistical techniques across 31.1M records to identify and remove redundant or uninformative features, reducing 186 features (CP4) to 112 production features (CP5a):

Table 3.12: Dimensionality Reduction Techniques (2015-2019)

Features Removed in CP4→CP5 (33 features):

• High Collinearity: YEAR, QUARTER (redundant with FL_DATE); DISTANCE_high_corr, DISTANCE_GROUP_high_corr (redundant with log_distance)
• High Cardinality: TAIL_NUM (116K unique values), FL_DATE (1,826 unique dates), prediction_utc, origin_obs_utc, asof_minutes (timestamp fields)
• Correlation >0.85: Various weather feature pairs (HourlyWetBulbTemperature/HourlyDewPointTemperature r=0.96)
• Zero Importance: Features showing <0.001 Gini importance in preliminary Random Forest models

Features Removed in CP5→CP5a (41 features, +12 indexed):

• Original String Columns: Removed 60 string columns after indexing (e.g., OP_UNIQUE_CARRIER, ORIGIN, DEST retained as indexed versions)
• Low Importance: 41 features with zero contribution in Random Forest importance analysis
• Added Indexed: 12 StringIndexed categorical features (carrier, airports, states, weather categories)
• Net Change: 153 → 112 features (-41 features, but gained 12 indexed versions of categoricals)

Key Highlights¶

1. Cyclic Encoding Rationale: Time and direction are circular variables where the distance between 23:59 and 00:01 should be 2 minutes, not 1,438 minutes. Sin/cos transformations preserve this topology for 14 temporal and directional features (7 pairs): dep_time, arr_time, day_of_week, month, plus wind direction.

2. Breiman's Stacked Generalization: Following Leo Breiman's methodology, we trained a Random Forest on CP3 features to generate probability predictions (rf_prob_delay) as meta-features in CP4. This allows linear models (Logistic Regression) to leverage complex non-linear decision boundaries learned by tree ensembles, effectively creating a two-stage ensemble.

3. Correlation-Based Selection: We removed 33 features with Pearson correlation >0.85 to reduce multicollinearity. When choosing between correlated pairs, we prioritized features with: (a) higher correlation with target DEP_DEL15, (b) better domain interpretability, (c) engineered aggregations over raw values (e.g., kept dep_delay15_24h_rolling_avg_by_origin_weighted over simple dep_delay15_rolling_avg).

4. Strict Temporal Validation: All transformations respect the T-2h prediction cutoff. Rolling windows use RANGE BETWEEN UNBOUNDED PRECEDING AND INTERVAL '2' HOUR PRECEDING to exclude same-flight information. RFM features use WHERE FL_DATE < current_flight_date to prevent look-ahead bias. Network features computed from historical flight patterns only.

5. Graph-Based Features: Airport network analysis using NetworkX generated 8 features capturing systemic delay propagation. PageRank identifies hub airports where delays cascade through connections. Degree centrality measures airport connectivity. Betweenness identifies critical transfer points. All metrics computed from 2015-2018 flight network for 2019 predictions.

Feature Engineering Progression by Stage¶

Table 3.13: Feature Count Evolution by Category (2015-2019)

Stage Transitions:

• CP2 (Base): 59 features after cleaning and leakage removal
• CP3 (Basic Features): 95 features (+36) - temporal, distance, weather, rolling aggregates
• CP4 (Advanced Features): 186 features (+91) - RFM, network, interactions, cyclic, Breiman, aircraft lag
• CP5 (Optimized): 153 features (-33) - correlation-based reduction, high-cardinality removal
• CP5a (Production): 112 features (-41, +12 indexed) - importance-based selection, string indexing

3.3.9 Feature Families and Data Dictionary¶

Feature Family Distribution¶

Feature Family Categorization¶

Table 3.14: Feature Families Summary (CP5a - 2015-2019)

Feature Composition Analysis¶

Raw Features (32 features, 29% of total):

• Temporal identifiers (7): FL_DATE, prediction_utc, origin_obs_utc, DAY_OF_MONTH, DAY_OF_WEEK, CRS_ARR_TIME, asof_minutes
• Flight identifiers (1): OP_CARRIER_FL_NUM
• Airport identifiers (4): ORIGIN, DEST, ORIGIN_AIRPORT_ID, DEST_AIRPORT_ID
• State identifiers (2): ORIGIN_STATE_ABR, DEST_STATE_ABR
• Weather measurements (8): HourlyDryBulbTemperature, HourlyDewPointTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyVisibility, HourlyRelativeHumidity, HourlyStationPressure, HourlyAltimeterSetting
• Geographic coordinates (6): origin_airport_lat, origin_airport_lon, dest_airport_lat, dest_airport_lon, origin_station_dis, dest_station_dis
• Categorical (2): origin_type, OP_UNIQUE_CARRIER (original string versions retained for validation)
• Target and reference (2): DEP_DEL15 (target), DEP_DELAY (reference)

Engineered Features (80 features, 71% of total) - By Engineering Method:

1. Rolling Window Aggregations (18): 24-hour weighted rolling averages by origin/carrier/day-of-week, same-day delay percentages, prior-day delay rates, 30-day volumes, congestion ratios, real-time delay counts
2. Cyclic Encodings (14): Sin/cos transformations for departure time (2), arrival time (2), day of week (2), month (2), wind direction (2), plus 4 additional temporal cyclic features
3. Interaction Terms (13): Multiplicative features capturing compounding effects between weather and delays, distance and peak hours, carrier and time, origin airport and weather conditions
4. Indexed Categoricals (12): Numeric indices for carrier, origin/destination airports, states, weather categories, season, airline reputation category, turnaround category, day-hour interaction, sky condition
5. Network/Graph Features (8): Airport centrality (degree, PageRank, betweenness), delay propagation scores, network cascade effects, historical 1-year delay rates
6. RFM Pattern Features (8): Days since last delay on route/carrier, 30-day route delay counts and rates, carrier delays at origin airport, recency-frequency-monetary proxy metrics
7. Distance Transformations (6): Log-transformed distance, categorical distance bins (very_short, short, medium, long, very_long), distance-based flags
8. Aircraft Lag Features (6): Previous flight delay status (prev_flight_dep_del15), turnaround time category, hours since previous flight, scheduled elapsed time, first-flight indicators, airport-wide delays
9. Temporal Indicators (3): Season categories, event flags (SuperBowl week, major events, airport maintenance, natural disasters), days since epoch
10. Weather Composites (3): Weather condition categories, sky condition parsed, temperature anomaly flags
11. Congestion Metrics (2): Airport traffic density, carrier flight count
12. Breiman Meta-Features (2): Random Forest probability predictions, binned probability categories
13. Reputation/Quality (2): Airline reputation score, airline reputation category
14. Miscellaneous Engineered (1): Day-hour interaction categories

Feature Engineering Insights¶

This 71% engineered feature composition (80 out of 112 features) reflects our core hypothesis: predictive power for flight delays emerges primarily from capturing operational patterns, temporal dependencies, network effects, and complex interactions rather than raw measurements alone.

3.3.10 Dataset Sizes and Storage¶

Storage Metrics Across Pipeline¶

Table 3.15: Dataset Sizes and Storage Requirements (2015-2019)

Storage Insights¶

Parquet Compression Performance:

• Average cell size ranges from 4-8 bytes, demonstrating excellent columnar compression across 31M+ records
• Compression ratio: Approximately 10-12x vs. raw CSV format
• String indexing in CP5a provides modest storage optimization (19.2 GB → 18.2 GB from CP5, -5%)
• Correlation-based feature removal in CP5 achieved significant savings (22.4 GB → 19.2 GB from CP4, -14%)

Storage Efficiency by Stage:

• Stage 0 → CP1: Consolidated 214 columns to 75 (-63%) with 18.5 GB output despite +74K rows
• CP1 → CP2: Row reduction (-618K) and column pruning (75→59) saved 33% storage (18.5 GB → 12.3 GB)
• CP2 → CP3: Basic features (+36 columns) increased size by 20% (12.3 GB → 14.8 GB)
• CP3 → CP4: Advanced features (+91 columns) increased size by 51% (14.8 GB → 22.4 GB)
• CP4 → CP5: Feature optimization (-33 columns) reduced size by 14% (22.4 GB → 19.2 GB)
• CP5 → CP5a: Importance filtering (-41 columns) provided 5% savings (19.2 GB → 18.2 GB)

Multi-Year Scaling Validation:

Computational Resource Requirements (Phase 3 Actual):

• Cluster configuration: 8-node Databricks cluster (Standard_DS3_v2: 4 cores, 14GB RAM per node)
• Driver node: 14GB RAM, 4 cores
• Executor nodes: 7 workers x 14GB RAM = 98GB total distributed memory
• In-memory operations: Feasible with strategic caching and unpersisting at each checkpoint
• Iterative modeling: Fast Parquet I/O enables rapid experimentation even on 31M records
• Weather join runtime: ~6 hours on 31.7M records with haversine distance calculations
• Total pipeline runtime: ~15 hours (Stage 0 → CP5a with all checkpoints)

Storage Location and Organization:

• Base directory: dbfs:/student-groups/Group_4_4/
• Checkpoint files: checkpoint_[1-5]_*_2015-2019.parquet (numbered stages)
• Production dataset: checkpoint_5_comprehensive_2015-2019_refined.parquet (modeling-ready)
• Metadata/reports: CSVs_5Y/ (analysis outputs, feature lists, validation reports)
• Visualizations: Charts_5Y/ (EDA plots, pipeline diagrams, correlation heatmaps)
• Raw source data: dbfs:/mnt/mids-w261/OTPW_60M_Backup/ (read-only, shared)

Memory Management Strategies for 31M Records:

1. Checkpoint-based workflows: Intermediate results saved at each stage prevent full pipeline recomputation on failures
2. Strategic caching: Frequently-accessed DataFrames (airport dimension, station bridge) cached with explicit unpersisting
3. Partition tuning: Adjusted Spark partitions from 200 (default) to 400-800 based on stage data volume
4. Broadcast joins: Small lookup tables (<1MB) broadcast to all executors for efficient joins
5. Lazy evaluation: Leveraged Spark's lazy execution to avoid materializing unnecessary intermediate DataFrames
6. Column pruning: Selected only required columns in joins and aggregations to minimize shuffle data
7. Predicate pushdown: Applied filters as early as possible to reduce data volume through pipeline

Scalability to Future Years (2020+):

The pipeline successfully scaled 6× from Phase 2 to Phase 3, validating that:

• Adding 2020-2021 data (~12M rows) would increase CP5a to ~25 GB (37% increase from 18.2 GB)
• Full 10-year dataset (2015-2024, ~62M rows) would require ~36 GB storage and 16-node cluster
• Current architecture supports extension to 10 years without fundamental redesign
• Databricks autoscaling can dynamically adjust resources based on workload

3.3.11 Final Dataset Validation and Production Readiness¶

Production Readiness Checklist¶

Table 3.16: Dataset Validation Results (CP5a - 2015-2019)

4 Exploratory Data Analysis (EDA):¶

4.1 Data Overview¶

%md

Insights¶

Correlation patterns show that system congestion is the primary driver of departure delays. Airport-wide delay counts, rolling origin delay ratios, and oncoming-flight volume exhibit the strongest relationships with DEP_DEL15, confirming that delays propagate through the network rather than occurring independently. Weather variables show modest linear correlations but clear monotonic patterns, meaning weather becomes a major amplifying force only when congestion is already high. Distance and schedule-based flight characteristics contribute minimal predictive value. A small set of engineered congestion and rolling-window features captures nearly all meaningful signal, reinforcing the importance of nonlinear models capable of learning thresholds and interactions.

Modeling Implications¶

• Prioritize congestion-based engineered features such as rolling averages, airport-wide delay metrics, and oncoming-flight counts.
• Use nonlinear models (GBT, Random Forest) that capture propagation effects and threshold behavior.
• Model weather interactions rather than relying solely on raw variables; monotonic patterns improve predictive strength.
• Reduce emphasis on distance and static schedule characteristics, which add limited incremental value.
• Keep the feature set focused on the 10–12 highest-signal engineered variables to improve stability and interpretability.

Business Implications¶

• Delays are system-driven, meaning operational interventions—not route changes—yield the highest ROI.
• Weather disruptions intensify under congestion, underscoring the need for early-day discipline and proactive buffer protection.
• Airport-level operational investments (gate management, staffing, runway flow programs) deliver stronger impact than schedule redesign.
• A compact high-signal feature set supports real-time delay-risk dashboards for operations leaders.
• Nonlinear delay dynamics show small disruptions escalate quickly, justifying tighter controls during peak-volume periods.

Busiest Origin Airports

The top 20 busiest airports account for the majority of U.S. departures. Despite heavy traffic, several major hubs (ATL, DEN, PHX, SEA) maintain relatively stable delay rates. This confirms that volume alone does not drive delay risk.

Most Delayed Origin Airports

When sorting by delay rate (n ≥ 5,000 flights), a different pattern emerges: several mid-volume airports (BWI, MDW, DAL, HPN) consistently exceed 20–25% delay rates. These structural delays persist across both time windows.

Modeling Implications

• Airport-specific context (weather, congestion, runway capacity) must be included.
• High-delay airports should receive explicit evaluation slices to prevent the model from over-generalizing patterns from high-volume hubs.
• The 3-month sample accurately reflects the full-year signal.

5. Machine Learning Algorithms (planned, scalable, leakage-aware):¶

Our task is to decide, two hours before scheduled departure, whether a flight will leave 15 minutes late or more (DEP_DEL15 = 1). In the 1-year OTPW sample this is clearly an imbalanced classification problem: roughly 1 in 5 flights is delayed, 4 in 5 are on time. That profile is the same one reported in most BTS/NOAA-based flight-delay papers, and in those studies tree and boosted models usually outperform simple linear models once you add schedule, airport, and weather features. Because we ultimately have to repeat this on the multi-year OTPW (tens of millions of rows), we are deliberately choosing algorithms that (i) work well on structured/tabular data such as “time of day, carrier, origin/dest, weather-as-of,” (ii) already exist in Spark/MLlib or have known distributed versions, and (iii) can be retrained once we replace the prejoined OTPW with our own flights → airport → station → weather joins. In other words, we want something that is good enough now, but won’t block us later when the data shape improves.

5.1 Machine Learning Approach¶

We have a leakage-free feature set that respects the T–2h rule, and a sensible classifier actually learns signal from it. To do that we will start with Logistic Regression in Spark. It is fast, easy to explain to an airline stakeholder, and it gives us clean probabilities that we can later threshold.

The logistic regression model predicts the probability of delay as:

$$ P(y=1 \mid \mathbf{x}) = \frac{1}{1 + e^{-(\beta_0 + \boldsymbol{\beta}^T \mathbf{x})}} $$

where x represents our feature vector and β are the learned coefficients.

On the data side, this model will only see features that are truly available at or before two hours prior to departure (scheduled times, carrier, origin/destination, calendar features such as day of week or month, plus the "as-of" weather features we already described). Because the target is imbalanced we will turn on class weights, which modifies the loss function to:

$$ L = -\frac{1}{n} \sum_{i=1}^{n} w_i \left[ y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i) \right] $$

where the weights are assigned as:

$$ w_i = \begin{cases} w_{\text{delayed}} & \text{if } y_i = 1 \\ w_{\text{ontime}} & \text{if } y_i = 0 \end{cases} $$

Alternatively, we will use threshold tuning so that the model does not collapse to "always on time." The primary metric we will report is F(0.5), not F₁, because in operations **raising a false alert is worse than missing a real delay (status quo) **;

Right after that we will keep a very dumb regression baseline just for orientation: for those flights that are actually delayed (or that the classifier tags as delayed) we will fit a plain Linear Regression on delay minutes:

$$ \text{Delay}_{\text{minutes}} = \alpha_0 + \boldsymbol{\alpha}^T \mathbf{x} + \epsilon $$

and we will compare it with an even simpler "predict the average delay" constant model:

$$ \hat{y} = \bar{y} = \frac{1}{n_{\text{delayed}}} \sum_{i: y_i = 1} \text{Delay}_{\text{minutes}, i} $$

This restores the original intent in our plan ("linear reg or average delay") and gives Phase 3 something to beat when we bring in boosted regressors.

We will run a small Random Forest (or shallow GBT) on the same leakage-free table. For Random Forest classification, we use Gini impurity as the splitting criterion:

$$ \text{Gini} = 1 - \sum_{i=1}^{C} p_i^2 $$

where C is the number of classes and p_i is the proportion of class i at a given node. For Gradient Boosting classification, we minimize cross-entropy loss:

$$ L = -\sum_{i=1}^{n} [y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i)] $$

The only purpose here is to show that the 12-month custom joined sample really contains nonlinear interactions — for example, that delays rise faster in the late afternoon at congested hubs under low visibility. RF/GBT on Spark have already been used at this scale in airline use cases, so we're staying inside the envelope.

5.1.1 Preferred approach: two-stage pipeline¶

For the actual project we prefer a two-stage design. This follows the pattern used in several recent flight-delay and transport-delay papers: first decide "will it be delayed?", and only then estimate "by how much?" The reason is simple: only about 20% of the flights are interesting from the delay-minutes point of view. If we train a regressor on everyone, 80% of the rows are "boring" and we waste compute on the cluster.

Formally, our two-stage approach can be expressed as:

$$ \text{Stage 1: } \hat{y}_{\text{binary}} = f_{\text{classifier}}(\mathbf{x}) $$

$$ \text{Stage 2: } \hat{y}_{\text{minutes}} = \begin{cases} g_{\text{regressor}}(\mathbf{x}) & \text{if } \hat{y}_{\text{binary}} = 1 \\ 0 & \text{if } \hat{y}_{\text{binary}} = 0 \end{cases} $$

In Stage 1 we will keep the binary classification task. We will start with Logistic Regression to have a transparent baseline, and then add a tree-ensemble model (Random Forest / GBT / XGBoost-style, depending on what we run first in Databricks). All of these will be trained with time-series-aware cross-validation — "train on earlier months, validate on later months" — to make sure we never peek at future weather or future airport congestion. The primary metric here remains F(0.5) (precision is more important than recall at this stage):

$$ F_{0.5} = 1.25 \cdot \frac{\text{Precision} \cdot \text{Recall}}{0.25 \cdot \text{Precision} + \text{Recall}} $$

and we will also show a confusion matrix by carrier and by origin airport so we can see if the model is only good on the big hubs.

In Stage 2 we will run a regression model only on the flights that Stage 1 flags as delayed. Here we will move to the usual suspects that do well on tabular data: gradient-boosted regressor, XGBoost regressor, or RF regressor. We will report MAE (because it is easy to read: "we're off by ~6 minutes"):

$$ \text{MAE} = \frac{1}{n_{\text{delayed}}} \sum_{i: \hat{y}_{\text{binary},i} = 1} \lvert y_{\text{minutes},i} - \hat{y}_{\text{minutes},i} \rvert $$

and RMSE (because most papers report it):

$$ \text{RMSE} = \sqrt{ \frac{1}{n_{\text{delayed}}} \sum_{i: \hat{y}_{\text{binary},i} = 1} (y_{\text{minutes},i} - \hat{y}_{\text{minutes},i})^2 } $$

The advantage of doing it this way is that we spend cluster time and feature-engineering effort only on the flights that actually need it.

So the story we want in the notebook is: Phase 2 proves the 2 step process works and is leakage-aware; Phase 3 adds the model tuning part.

5.1.2 Cross Validation¶

The method that we used for cross-validating the time-series model is cross-validation on a rolling basis. We started with a small subset of data for training purposes, forecast for the later data points, and then check the accuracy of the forecasted data points. The same forecasted data points are included as part of the next training dataset, and subsequent data points are forecasted. We will use a rolling window of fixed size because we have a very large dataset. For small datasets, it may be appropriate to use expanding window for cross validation.

For Cross Validation, first, we will split the dataset based on time upfront, train data for first 6 months and test data for last 3 months. Next, we further split the dataset during the cross validation on a per fold basis. For each iteration, 6 months of data will be used for training and 1 month will be used for testing. So, for 1 year data we will use first 9 months as train data and last 3 months data for evaluation.

raw_data → [Feature Selection] → [Encoding] → [Scaling] → preprocessed_data → [Time Series CV]

5.2 Alternative: 2 Stage Architecture¶

The proposed 2 Stage Architecture proposed in Phase 1 did not show good performance for the second stage. This was because the second stage was only trained on the delayed flights. It was unable to learn the flight delay pattern. So, in Phase 2, we modified this architecture to train the Stage 2 regressor on all of the data so the model is able to learn the flight pattern better. In Phase 3, we attempted different ways of combining the results from the classifier and regressor to provide the best possible prediction on the delay value.

Our flight delay prediction approach evolved through several phases, progressively simplifying the architecture while improving performance. We initially implemented a two-stage pipeline combining a classifier (Logistic Regression, RandomForest, GBTClassifier, or SparkXGBClassifier) to predict whether a delay would occur, followed by a regressor (Linear Regression, GBTRegressor, or SparkXGBRegressor) to predict delay duration, experimenting with various ensemble strategies including sequential filtered training, threshold-gated prediction, and probability-weighted combinations. Finding that the classifier provided minimal benefit, we transitioned to a regression-only ensemble approach using two SparkXGBRegressor models—one trained with sample weights (1x/2x/2.5x for delays ≤60min/60-120min/>120min) to emphasize severe delays, and one without weights for balanced predictions—combined using Max, Min, or Average strategies, with deeper trees (max_depth=11) and regularization (reg_alpha=0.2, reg_lambda=1.0) to prevent overfitting. The Max ensemble achieved the best RMSE (41.69 minutes) by taking the higher prediction from both models, effectively capturing severe delays, while the Min ensemble achieved the best MAE (11.92 minutes) by optimizing for typical cases. Finally, to enable binary classification evaluation against the DOT standard (DEP_DEL15), we converted regression outputs to binary predictions using a 15-minute threshold, achieving F1=0.668, F2=0.697, and AuPRC=0.722, demonstrating that a well-tuned regression model can effectively serve both continuous delay prediction and binary delay classification tasks without requiring a separate classifier.

%md

5.2.1 Summary of 2 Stage and Regression only Experiments¶

NOTE: Phase 3 Experiments were conducted on a cluster with following configuration: 8× m5d.2xlarge (32GB, 8 cores)

Summary of Regression only Experiments¶

Key Findings¶

Desciption of terms used in table:

• Sequential (Filtered Training): Regressor trained only on delayed samples
• Sequential (Filtered Inference): Regressor trained on all data, predictions gated by classifier
• Threshold-Gated prediction: Returns the predicted delay value when the probability exceeds the specified threshold, otherwise returns zero
• Probability-weighted: P(delay) × predicted value
• Weighted Model : Regression Model trained using weights on delay value

Key Findings¶

Best Performers (Test RMSE)¶

Best Performers (Test MAE)¶

Best Performers (Binary Classification - vs DEP_DEL15)¶

Evolution Across Phases¶

Key Insights¶

1. Ensemble Max achieves best RMSE (41.69): Combining weighted and unweighted XGBoost models with Max strategy reduces RMSE by 2.7% vs previous best
2. Ensemble Min achieves best MAE (11.92): Taking the lower prediction optimizes for average-case performance
3. Trade-off between RMSE and MAE: Max strategy favors RMSE (severe delays), Min strategy favors MAE (typical delays)
4. Sample weighting improves generalization: Weighting severe delays (2x-2.5x) helps capture extreme cases
5. Deeper trees with regularization: max_depth=11 with reg_alpha=0.2, reg_lambda=1.0 prevents overfitting
6. Two-stage classifier no longer needed: Direct regression with ensemble outperforms classifier + regressor pipeline
7. Binary classification from regression: Converting regression output to binary achieves F2=0.697, AuPRC=0.722

Recommended Model Configuration¶

Model Parameters (Regression only)¶

# XGBoost Configuration
max_depth = 11
learning_rate = 0.05
n_estimators = 200
reg_alpha = 0.2
reg_lambda = 1.0
subsample = 0.8
colsample_bytree = 0.8

# Sample Weights (weighted model only)
weight = 1.0  # if delay ≤ 60 min
weight = 2.0  # if 60 < delay ≤ 120 min
weight = 2.5  # if delay > 120 min

# Ensemble Strategies
Max: max(pred_weighted, pred_unweighted)  # Best for RMSE
Min: min(pred_weighted, pred_unweighted)  # Best for MAE
Avg: (pred_weighted + pred_unweighted) / 2  # Balanced

5.2.2 Regression Value to Binary Classification Results Summary¶

Overall Model Performance¶

The ensemble model achieves 86.1% accuracy in predicting whether a flight will be delayed by 15 minutes or more (DEP_DEL15). This is a strong result given the inherent unpredictability of flight delays.

Core Metrics Interpretation¶

Confusion Matrix Analysis¶

                      Predicted
                   No Delay  |  Delay
Actual No Delay:  5,268,354  |  639,191   (89.2% correct)
Actual Delay:       372,284  |  979,178   (72.5% correct)

Key Observations:

• True Negatives (5.27M): Correctly predicted on-time flights
• True Positives (979K): Correctly predicted delays
• False Positives (639K): Predicted delay but flight was on-time (unnecessary alerts)
• False Negatives (372K): Missed delays (passengers caught off-guard)

Ensemble Strategy Comparison¶

Key Finding: There's a clear trade-off:

• Weighted model / Max ensemble: Better at catching delays (high recall: 72.5%)
• Unweighted model / Min ensemble: Fewer false alarms (high precision: 71.8%)
• Average ensemble: Best balance (highest F1: 66.8%)

Optimal Threshold Analysis¶

The default threshold of 15 minutes (matching DEP_DEL15 definition) may not be optimal:

Recommendations:

• Use 8-minute threshold if priority is catching delays (F2 optimized)
• Use 18-minute threshold for best overall balance (F1 optimized)
• Use 25+ minute threshold if false alarms are costly (precision optimized)

Practical Implications¶

For Airlines/Operations:

• Model catches 72.5% of delays before they happen
• 27.5% of delays are missed (372K flights) - room for improvement
• 639K false alarms - may cause unnecessary resource allocation

For Passengers:

• If model predicts delay: 60.5% chance it will actually be delayed
• If model predicts on-time: 93.4% chance it will be on-time (high NPV)

Summary Statement¶

The regression-based ensemble model, when converted to binary delay prediction, achieves 86% accuracy with a 72.5% recall rate for catching delays. The weighted model excels at identifying delays (fewer missed), while the unweighted model excels at precision (fewer false alarms). For operational use, an 18-minute prediction threshold provides the best F1 balance, while an 8-minute threshold maximizes delay detection at the cost of more false positives.

5.2.3 Error Analysis¶

Delay Distrubtion by Value¶

Weighted Model Performance by Delay Magnitude¶

The model shows a clear asymmetric error pattern: it overpredicts small delays and severely underpredicts large delays.

Performance Breakdown¶

Well-Calibrated Range (11-30 min delays):

Overprediction Zone (0-10 min delays):

Severe Underprediction Zone (31+ min delays):

Key Takeaways¶

1. Sweet spot exists: Model performs best for delays between 11-30 minutes (near-zero bias)

2. Regression to the mean: Model predictions cluster around 30-65 minutes regardless of actual delay severity

3. Extreme delays are problematic: For delays >3 hours, model only predicts ~66 minutes - missing by 4+ hours on average

4. Conservative predictions: Model appears to "cap" predictions around 65 minutes, unable to capture tail events

5. Practical impact:

   • For passengers: Minor delays ( less than 15 min) are not conidered as actual delay. So minimal impact
   • For operations: Cannot rely on model for severe delay scenarios

Recommendation¶

Consider a separate model or adjustment factor for severe delays (>45 min), or implement prediction intervals that widen for longer predicted delays.

Worst Airports and Carriers by error¶

Key Insights Summary¶

Worst Airports¶

Worst Carriers¶

Pattern: Regional carriers and small/mountain airports are hardest to predict - likely due to weather sensitivity and fewer training samples.

Best and Worst Routes by error¶

Top 10 Routes: Actual vs Predicted Delay Summary¶

The model consistently underpredicts delays across all top 10 routes, with gaps ranging from -2.4 to -4.5 minutes.

Route-by-Route Insights¶

Highest Actual Delays:

• ORD-UA (United at O'Hare): ~17.7 min actual, largest delay among top routes
• DFW-AA (American at Dallas): ~16.9 min actual, second highest
• MDW-WN (Southwest at Midway): ~15.2 min actual

Lowest Actual Delays:

• ATL-DL (Delta at Atlanta): ~9.5 min actual, efficient hub operation
• SEA-AS (Alaska at Seattle): ~9.5 min actual, well-run hub
• MSP-DL (Delta at Minneapolis): ~10.5 min actual

Prediction Gap Analysis¶

Key Takeaways¶

1. Southwest (WN) routes are better calibrated: DEN-WN (-2.4), LAS-WN (-2.8) have smallest gaps
2. American Airlines (AA) routes have largest errors: DFW-AA and CLT-AA both at -4.5 minutes
3. High-volume Delta hubs perform well: ATL-DL has low delay and small gap (-2.7)
4. Systematic underprediction: No route shows overprediction - model bias is consistent

Weekend vs Holiday comparison¶

Performance Analysis Summary¶

Weekend × Holiday Analysis¶

Key Finding: Holiday periods reduce both actual delays and prediction errors.

Observations:

• Holiday effect is stronger than weekend effect: Holiday months reduce actual delays by ~1.5-2 minutes regardless of weekend status
• Model consistently underpredicts: All segments show 5-6 minute underprediction gap
• RMSE is relatively stable: Ranges from 41.7 to 44.3 across all segments
• Best performance: Weekend + Holiday combination (lowest MAE of 10.7 minutes)

Top Routes (Origin + Carrier) Analysis¶

Best Performing Routes:

• LAS-WN (Southwest at Las Vegas): Lowest RMSE (21.1), MAE of 9.5 minutes
• DEN-WN (Southwest at Denver): RMSE of 24.1, well-calibrated predictions
• PHX-WN (Southwest at Phoenix): RMSE of 22.7, consistent performance

Worst Performing Routes:

• ORD-OO (SkyWest at O'Hare): Highest RMSE (54.6), highest actual delay (20.7 min)
• ORD-UA (United at O'Hare): High RMSE (42.2), high actual delay (17.7 min)
• DEN-UA (United at Denver): RMSE of 40.7, significant underprediction

Key Patterns:

1. Southwest (WN) routes perform best: Consistently lower RMSE across LAS, DEN, PHX, MDW, DAL, BWI - likely due to point-to-point operations and predictable patterns

2. O'Hare (ORD) is challenging: Multiple carriers (OO, UA, AA) show high errors at ORD - hub complexity and weather issues

3. Regional carriers struggle: SkyWest (OO) at ORD has the worst performance (RMSE 54.6) - regional operations are harder to predict

4. Delta hubs perform well: ATL-DL has the highest volume (243K) with reasonable RMSE (28.1) - efficient hub operation

5. Consistent underprediction: All routes show predicted delays 3-6 minutes below actual - systematic bias in the model

5.2.4 Feature Importance for SparkXGB Regressor and Classifier¶

Top 20 Feature Importances Summary¶

The top 20 features capture 77.7% of total model importance, demonstrating that a relatively small subset of features drives most of the predictive power. The steep initial rise followed by a flattening curve shows:

• First 5 features: Capture ~55% of importance
• First 10 features: Capture ~68% of importance
• First 20 features: Capture ~78% of importance

This indicates potential for dimensionality reduction - a model using only the top 20-40 features may perform nearly as well as one using all features, with faster training and reduced overfitting risk.

Top 3 features account for ~45% of importance:

1. prev_flight_dep_del15 (~32%): Whether the previous flight was delayed is by far the strongest predictor - indicating delay propagation is the primary driver
2. num_airport_wide_delays (~8%): Current airport congestion level
3. prior_day_delay_rate (~5%): Historical delay patterns from the previous day

Feature Categories¶

Delay History Features (dominant):

• Previous flight delay status
• Prior day delay rate
• Days since last delay on route
• Rolling delay averages

Airport/Operational Features:

• Airport-wide delay counts
• Delay propagation score
• Rolling 30-day volume
• Origin betweenness (network centrality)

Temporal Features:

• Hours since previous flight
• Departure time (sin/cos encoded)
• Arrival time (cos)
• Day of week interactions

Route/Carrier Features:

• Destination encoded
• Route delay rate (30-day)
• Carrier-weighted rolling averages

Feature Importance Comparison: Classifier vs Regressor¶

1. Classifier focuses on binary signals: The classifier heavily relies on whether the previous flight was delayed - a strong binary indicator for predicting if a delay will occur.

2. Regressor uses more diverse inputs: The regressor distributes importance across more features (prior_day_delay_rate, days_since_last_delay_route, rolling averages) because predicting how long a delay will be requires more nuanced information.

3. Shared important features: Both models value:

   • Previous flight delay status
   • Airport-wide delay counts
   • Rolling delay averages
   • Delay propagation metrics

Dominant Feature: Both models agree that prev_flight_dep_del15 (whether the previous flight was delayed) is the most important predictor, but with different magnitudes:

• Classifier: ~45% importance (heavily dominant)
• Regressor: ~32% importance (still dominant but less extreme)

Second Most Important: Both models rank num_airport_wide_delays as the second most important feature (~10% for both).

Key Differences¶

Implication for Ensemble¶

The different feature emphasis suggests the classifier and regressor capture complementary information - supporting their use together in a two-stage prediction pipeline.

Top 20 Feature Importances Summary¶

The top 20 features capture 77.7% of total model importance, demonstrating that a relatively small subset of features drives most of the predictive power.

Dominant Features¶

Top 3 features account for ~45% of importance:

1. prev_flight_dep_del15 (~32%): Whether the previous flight was delayed is by far the strongest predictor - indicating delay propagation is the primary driver
2. num_airport_wide_delays (~8%): Current airport congestion level
3. prior_day_delay_rate (~5%): Historical delay patterns from the previous day

Feature Categories¶

Delay History Features (dominant):

• Previous flight delay status
• Prior day delay rate
• Days since last delay on route
• Rolling delay averages

Airport/Operational Features:

• Airport-wide delay counts
• Delay propagation score
• Rolling 30-day volume
• Origin betweenness (network centrality)

Temporal Features:

• Hours since previous flight
• Departure time (sin/cos encoded)
• Arrival time (cos)
• Day of week interactions

Route/Carrier Features:

• Destination encoded
• Route delay rate (30-day)
• Carrier-weighted rolling averages

Cumulative Curve Interpretation¶

The steep initial rise followed by a flattening curve shows:

• First 5 features: Capture ~55% of importance
• First 10 features: Capture ~68% of importance
• First 20 features: Capture ~78% of importance

This indicates potential for dimensionality reduction - a model using only the top 20-40 features may perform nearly as well as one using all features, with faster training and reduced overfitting risk.

5.2.5 Summary for 2 Stage Ensemble Experiments¶

Summary of 2-Stage Experiment Evolution¶

Phase 1: Baseline Approach¶

The initial experiment used simple models (Logistic Regression + Linear Regression) with class weights to handle imbalance. This approach performed poorly with a test RMSE of 97.49 minutes and MAE of 53.59 minutes, establishing a baseline for improvement.

Phase 2: Tree-Based Models¶

Switching to tree-based models (RandomForest + GBTRegressor) with undersampling at 0.5 ratio showed promise in training (RMSE: 19.43) but suffered from severe overfitting, with test RMSE jumping to 74.22 minutes. The large train-test gap indicated the model wasn't generalizing well.

Phase 3: Scaled Training & Two-Stage Pipeline¶

Using the full dataset (2015-2018) for training and 2019 for testing dramatically improved results. Several key findings emerged:

Model Comparison:

• SparkXGBoost consistently outperformed RandomForest and GBTClassifier combinations
• XGBoost achieved test RMSE of 42.83 and MAE of 12.07-12.30

Ensemble Strategy Comparison:

• Regression-only achieved RMSE of 42.83 min - classifier gating provided minimal benefit
• Threshold-Gated performed nearly identically (42.85 min RMSE)
• Probability-weighted achieved MAE of 11.66 min but higher RMSE (44.14 min)

Balance Strategy:

• Increasing undersample ratio from 0.5 to 1.0 improved performance
• Full balancing helped the model better learn delay patterns

Regression-Only Ensemble with Sample Weighting¶

Recognizing that the two-stage classifier-regressor pipeline provided minimal benefit, we eliminated the classifier and focused on optimizing the regression model with ensemble strategies:

Sample Weighting Strategy:

• Applied instance weights to emphasize severe delays during training
• Weight scheme: 1.0× for delays ≤60 min, 2.0× for 60-120 min, 2.5× for >120 min
• Weighted model better captured extreme delay patterns

Model Configuration:

• Deeper trees (max_depth=11) with regularization (reg_alpha=0.2, reg_lambda=1.0)
• Learning rate=0.05, n_estimators=200, subsample=0.8, colsample_bytree=0.8

Ensemble Strategies: Two XGBRegressor models (one with sample weights, one without) combined using:

• Max: Takes higher prediction → Best for RMSE (captures severe delays)
• Min: Takes lower prediction → Best for MAE (optimizes typical cases)
• Average: Balanced approach → Best for F1 score

Regression only Results:

Regression-to-Binary Classification Evaluation¶

Converting regression outputs to binary predictions (delay ≥15 min) enabled direct comparison with classification models:

Binary Classification Results (vs DEP_DEL15):

Key Takeaways¶

1. Regression-to-binary outperforms direct classification: XGBoost Ensemble (Max) achieves highest F2 (0.697) and AuPRC (0.723) across all approaches
2. Two-stage pipeline unnecessary: Classifier + Regressor provides minimal benefit over regression-only
3. Sample weighting improves severe delay prediction: Weighted model captures extreme delays better
4. Ensemble strategy depends on use case:
   • Maximize recall / F2: Use Ensemble Max (catches more delays)
   • Maximize precision / accuracy: Use Ensemble Min or Unweighted (fewer false alarms)
   • Balanced performance: Use Ensemble Average (best F1)
5. Training efficiency: XGBoost ensemble trains 15× faster than CNN while achieving better F2 and AuPRC
6. Overall improvement: From Phase 1 to Phase 3, test RMSE improved by 57% (97.49 → 41.69 minutes)

Recommended Model Configuration¶

Error Analysis Insights¶

By Delay Severity:

• Model performs well for mild delays (<30 min): ~80% recall
• Performance degrades for severe delays (>2 hr): ~50% recall
• Sample weighting helps but extreme delays remain challenging

By Carrier:

• Best: Hawaiian (HA), Alaska (AS), Delta (DL) - >89% accuracy
• Worst: ExpressJet (EV), SkyWest (OO), Mesa (YV) - <83% accuracy
• Regional carriers are hardest to predict

By Time/Season:

• Summer months (Jun-Aug) have highest error rates
• Morning flights more predictable than evening flights
• Weekends slightly harder to predict than weekdays

5.3 Tree Ensemble (Classification)¶

5.3.1 Loss Function¶

5.3.1.1 Gradient Boosted Trees (GBT) Loss Function¶

The GBT model minimizes a logistic loss function with gradient-based optimization: $$ L(y, F(x)) = \log\left(1 + e^{-2yF(x)}\right) $$ where:

• y ∈ {-1, +1} is the true label (mapped from 0/1)
• F(x) = sum from m=1 to M of (gamma_m times h_m(x)) is the ensemble prediction
• h_m(x) is the m-th decision tree (weak learner)
• gamma_m is the learning rate-adjusted weight (stepSize parameter)

Gradient Update at iteration m:

$$h_m(x) = \arg\min_{h} \sum_{i=1}^{N} \left( -\frac{\partial L(y_i, F_{m-1}(x_i))}{\partial F_{m-1}(x_i)} - h(x_i) \right)^2$$

Regularization Update:

$$F_m(x) = F_{m-1}(x) + \text{stepSize} \cdot h_m(x)$$

Regularization in GBT:

• Learning rate (stepSize): Controls contribution of each tree
• Subsampling (subsamplingRate): Randomly samples fraction of data for each tree
• Tree depth (maxDepth): Limits complexity of individual trees
• Early stopping: Monitors validation performance to prevent overfitting

5.3.1.2 Random Forest Loss Function¶

The Random Forest model optimizes a Gini impurity criterion for each decision tree in the ensemble:

Data Loss (Gini Impurity): $$ \text{Gini}(t) = 1 - \sum_{k=1}^{K} p_k^2 $$ where:

• t is a node in the decision tree
• K is the number of classes (2 for binary classification)
• p_k is the proportion of samples of class k at node t

Split Quality Measure:

$$ \Delta \text{Gini}(s, t) = \text{Gini}(t) - \left( \frac{N_{\text{left}}}{N} \text{Gini}(t_{\text{left}}) + \frac{N_{\text{right}}}{N} \text{Gini}(t_{\text{right}}) \right) $$

where:

• s is a candidate split
• N is the total number of samples at node t
• N_left and N_right are the numbers of samples in left and right child nodes

Regularization in RF:

• maxDepth parameter (limits tree depth)
• maxBins parameter (limits feature granularity)
• Random feature subset selection (featureSubsetStrategy="sqrt")
• Bootstrap sampling for each tree

5.3.2. Early Stopping Discussion¶

Data Used for Early Stopping¶

• Validation Set: 2018 flight data (df_val_2018)
• Training Set: 2015-2017 undersampled flight data
  • Undersampling ratio: delayed-to-on-time = 1:2 (33.3% delayed, 66.7% on-time)
  • Original train set: 17.9% delayed → Undersampled: 33.3% delayed
  • Total rows: 9,009,126
• Test Set (Blind): 2019 flight data (held out until final evaluation)

Metrics Used¶

ML Metrics:

• Primary metric: AUC-PR (Area Under Precision-Recall Curve)
• Secondary metrics: AUC-ROC, F0.5 score, Precision, Recall

Rationale: AUC-PR is more informative than AUC-ROC for imbalanced datasets. It directly measures the trade-off between precision and recall across different classification thresholds.

Business Metric Consideration: F0.5 score weighs precision twice as heavily as recall, reflecting the business context where false positives (incorrectly predicting a delay) are more costly than false negatives (missing a delay prediction).

5.3.2.1 Early Stopping Strategy for GBT¶

Approach: Progressive stopping with patience=2 and min_delta=0.001.

Implementation:

if auc_pr > best_score + min_delta:
    best_score = auc_pr
    no_improve_rounds = 0
else:
    no_improve_rounds += 1

if no_improve_rounds >= patience:
    print(f"Early stopping triggered at numIters={num_iters}")
    break

Decision Logic:

• We do NOT quit at the first sign of poor performance
• We allow 2 consecutive rounds without improvement before stopping
• Improvement is defined as AUC-PR gain > 0.001 (0.1%)
• This gives the model more patience to explore the iteration space

GBT Hyperparameter Grid:

• num_iter_grid: [70, 80, 90]
• max_depth: [3, 5]
• step_size: [0.05, 0.1]
• subsampling_rate: 0.8

Example from GBT Experiments: All three configurations completed all iterations without early stopping, indicating steady improvement throughout.

5.3.2.2 Early Stopping Strategy for RF¶

Approach: Progressive stopping with tolerance threshold of 0.001 in AUC-PR improvement.

Implementation:

if auc_pr > last_best_for_this_depth + 0.001:
    last_best_for_this_depth = auc_pr
    no_improve_rounds = 0
else:
    no_improve_rounds += 1

if no_improve_rounds >= 1:
    print(f"  -> early stop on numTrees for depth={maxDepth}")
    break

Decision Logic:

• We allow one round without improvement before stopping
• More aggressive than GBT (patience=1 vs patience=2)
• Rationale: RF trees are independent, so adding more trees without improvement is less likely to help

RF Hyperparameter Grid:

• numTrees_grid: [10, 15, 20]
• maxDepth_grid: [5, 8, 10, 15]

Example from RF Experiments:

• depth=5: Stopped after numTrees=15 (no improvement from 10→15)
• depth=8: Stopped after numTrees=15 (no improvement from 10→15)
• depth=10: Stopped after numTrees=15 (no improvement from 10→15)
• depth=15: Continued through numTrees=20 (steady improvement)

5.3.2.3 How Early Stopping Helps¶

1. Computational Efficiency: Saves training time by avoiding unpromising hyperparameter configurations
2. Prevents Overfitting:
   • GBT: Stops adding boosting iterations when validation performance plateaus
   • RF: Stops adding trees when ensemble diversity no longer improves performance
3. Resource Optimization: Reduces cluster usage for large-scale experiments (saved ~3 RF experiments)
4. Informed Search: Guides hyperparameter search toward more promising regions

5.3.3 Cross-Fold Validation Metrics¶

Status: Cross-fold validation was not implemented in this phase.

Current Approach:

• Single train/validation/test split
• Training: 2015-2017 (undersampled to 33.3% delayed, 66.7% on-time)
• Validation: 2018 (natural distribution: ~18% delayed)
• Test: 2019 (natural distribution: ~18% delayed)

Rationale for Skipping Cross-Validation:

1. Temporal Nature: Time-series data should not be shuffled; future data must not leak into training
2. Computational Cost: Cross-fold validation on 9M+ training samples with GBT (iter=90) and RF (depth=15, numTrees=20) would require 3-5× the compute time
3. Validation Strategy: Chronological split (2015-2017 → 2018 → 2019) better reflects real-world deployment scenario

5.3.4 Comprehensive Experiment Table¶

5.3.4.1 Gradient Boosted Trees Experiments¶

GBT Configuration Details:

• maxBins: 64 (all experiments)
• seed: 42 (all experiments)
• Training data: 2015-2017 undersampled (9,009,126 rows)

GBT Best Validation Config:

• maxDepth=5, stepSize=0.1, subsamplingRate=0.8, maxIter=90
• Val AUC-PR=0.7191, Val AUC-ROC=0.8945, Val F0.5=0.6415

5.3.4.2 Random Forest Experiments¶

RF Configuration Details:

• seed: 42 (all experiments)
• Training data: 2015-2017 undersampled (9,009,126 rows)